Selective allocation of processing resources for processing image data

ABSTRACT

In general, techniques are described regarding selective allocation of processing resources for processing image data. Processing circuitry may warp a frame of image data around variously indicated areas of the frame to create a warped frame of image data. The processing circuitry may allocate more pixels to the indicated areas by virtue of the frame warping operation. The warped frame may then be output for further processing in accordance with proportions of the warped frame resulting in better image or video quality in the indicated areas.

TECHNICAL FIELD

The disclosure relates to image processing.

BACKGROUND

Image capture devices are commonly incorporated into a wide variety ofdevices. In this disclosure, an image capture device refers to anydevice that can capture one or more digital images, including devicesthat can capture still images and devices that can capture sequences ofimages to record video. By way of example, image capture devices maycomprise stand-alone digital cameras or digital video camcorders,camera-equipped wireless communication device handsets, such as mobiletelephones having one or more cameras, cellular or satellite radiotelephones, camera-equipped personal digital assistants (PDAs), panelsor tablets, gaming devices, computer devices that include cameras, suchas so-called “web-cams,” or any devices with digital imaging or videocapabilities.

Certain image capture devices may include multiple image sensors thatcapture image data and transfer the image data to a camera processor.The image sensor may perform various techniques, such as pixel binning,prior to transferring pixels to the camera processor for processing. Thecamera processor may perform various pixel processing techniques, suchas scaling at various levels of scaling, digital cropping, performingstatistics algorithms, denoising, sharpening, etc. and apply variousother processing resources to a frame of image data. In particular, thecamera processor may perform various pixel processing techniques thatachieve a desired resolution. For example, the camera processor mayperform scaling of the image data using a scaling level that achieves adesired output data resolution. Scaling levels may include scalingratios that provide upscaling, downscaling, or in some cases, noscaling, such as at the threshold between downscaling and upscaling.

SUMMARY

In general, this disclosure describes image processing techniquesinvolving digital cameras having image sensors and camera processors.Particularly, a camera processor may be configured to pre-process aframe of image data that the camera processor receives from an imagesensor. The camera processor, or in some instances, an encoder/decoder,may then non-uniformly process certain areas of the pre-processed frameusing different processing resources relative to other areas of theframe.

In some examples, the pre-processing operation includes warping theframe of image data to create a frame that is warped around variousindicated areas of the frame. Specifically, the camera processor maypre-process the frame of image data according to indications ofparticular areas of a frame received or determined by the cameraprocessor. The indications indicate certain areas of the frame thatinclude regions of interest, objects of interest, etc., or otherdistinctive areas of the frame that are to be allocated more processingresources (e.g., more pixels). For example, the camera processor mayreceive, via automatic detection and/or user selection, indication thata particular area of a frame includes an area of importance. The cameraprocessor may then warp the frame around the indicated area, such thatvarious degrees of importance of the frame may be defined by warpedproportions of the warped frame.

In some examples, a camera processor may warp a frame that includes oneor more indicated areas to create the warped frame in a real-timeprocessing operation. For example, the camera processor may warp theframe in real-time as the frame of image data is received from an imagesensor by using an image front end (IFE) processor. In one example, acamera processor may apply a warping grid to one or more frames of imagedata to create the warped frame. In any event, by pre-processing theimage data by warping the frame around a user-defined or automaticallydetected area of a frame, such indicated areas may be allocated morearea of the warped frame and more pixels of the warped frame.

In some examples, the IFE may distort a frame of image data so as tocause the frame to warp around the indicated area and thereby allocatemore area of the frame to the indicated area. In such examples, warpinga frame of image data around an indicated area causes the indicated areato be enlarged relative to other areas of the frame. That is, warpingthe frame causes the indicated area to be represented by more pixels,while other areas of the image have fewer pixels. Thus, the indicatedarea may be allocated more processing resources than the indicated areawould be allocated in an unwarped configuration.

In this way, the camera processor may allocate more processing resourcesto variously indicated areas of the frame based on the pre-processingcreation of the warped frame. In some examples, the warped frame maythen be processed offline, output to memory, or output to anencoder/decoder for further processing. For example, an encoder/decoderor offline processor may perform processing operations that unwarp thewarped frame. In an illustrative example, the encoder/decoder or offlineprocessor may perform digital scaling of the warped frame to achieve adesired output resolution or to comply with limited processingcapabilities of the camera system.

In accordance with various techniques of this disclosure, the cameraprocessor may selectively apply various levels of scaling in amountsproportional to the amount of area of the frame allocated to the one ormore indicated areas through application of the pre-processing warpingoperation. The warping operation is applied around indicated areas,including user-indicated areas of importance and/or automaticallyrecognized areas of importance, such that more area of a frame isallocated the one or more indicated areas in creating a warped frame.Depending on the type of scaling used, the camera processor(s) may applymore upscaling, less downscaling, or in some instances, no scaling, withrespect to pixels corresponding to the one or more indicated area(s) ofa frame compared to areas of the frame adjacent the indicated area(s),such as background scenery. As such, the camera processor(s) mayallocate more pixels to the one or more indicated area(s) of the frameto be utilized for post-processing operations (e.g., scaling, denoising,sharpening, etc.) relative to other areas of the frame.

Scaling is one example of a processing resource that a camera processormay apply non-uniformly to a warped frame of image data. Selectiveallocation of such processing resources may be advantageous forproducing high quality images by reducing the amount of downscaling usedto process variously indicated areas of a frame. That is, reducing theamount of downscaling performed for an indicated area may improveoverall image quality because downscaling tends to degrade resultingimage or video quality. In an illustrative example, the camera processormay perform upscaling or no scaling at a first scaling level on certainareas of the warped frame and may perform downscaling at a secondscaling level on other adjacent areas of the frame in order to achieve adesired output resolution. In addition, camera processor may improveimage quality by performing upscaling of a warped frame prior toperforming other processing techniques, such as temporalsuper-resolution processing, sharpening, etc.

As such, a camera processor may selectively allocate processingresources (e.g., scaling, denoising, sharpening, etc.) according toproportions of a frame warped around one or more indicated areas of theframe (e.g., specific spatial regions or objects/people that may betracked across multiple frames). In this way, an indicated area maydynamically receive higher resolution compared to the areas surroundingthe indicated area because the indicated area is enlarged when the frameis warped, such that the camera processor allocates more area of theframe to the enlarged area. The output warped frame may include an areathat is allocated more pixels, such that the camera processor (e.g., animage processing engine) may be enabled to allocate more time andresources to processing the indicated areas relative to other areas ofthe frame.

In one example, the techniques of the disclosure are directed to anapparatus configured for camera processing, the apparatus comprising: amemory configured to store image data, and one or more processors incommunication with the memory, the one or more processors configured to:receive, from an image sensor, a frame of the image data; receiveindication of a first area corresponding to at least a portion of theframe; warp the frame around the first area to create a warped frame;and output the warped frame for processing.

In another example, the techniques of the disclosure are directed to amethod for camera processing, the method comprising: receiving, from animage sensor, a frame of image data; receiving indication of a firstarea corresponding to at least a portion of the frame; warping the framearound the first area; and outputting the warped frame for processing.

In another example, the techniques of the disclosure are directed to anapparatus configured for camera processing, the apparatus comprising:means for receiving a frame of image data; means for receivingindication of a first area corresponding to at least a portion of theframe; means for warping the frame around the first area to create awarped frame; and means for outputting the warped frame for processing.

In another example, the techniques of the disclosure are directed to anon-transitory computer-readable storage medium having stored thereoninstructions that, when executed, cause one or more processors to:receive a frame of image data; receive indication of a first areacorresponding to at least a portion of the frame; warp the frame aroundthe first area to create a warped frame; and output the warped frame forprocessing.

The details of one or more examples are set forth in the accompanyingdrawings and the description below. Other features, objects, andadvantages will be apparent from the description, drawings, and claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a device configured to perform one or moreof the example techniques described in this disclosure.

FIG. 2 is a block diagram illustrating example components of a cameraincluding camera processor(s) of FIG. 1, in accordance with variousaspects of the techniques described in this disclosure.

FIG. 3 is a conceptual diagram illustrating example real-time processingand offline processing of the camera processor(s) shown in FIG. 1 or 2,in accordance with various aspects of the techniques described in thisdisclosure.

FIG. 4 is an example flowchart illustrating an example method andtechnique of camera processor(s) shown in FIG. 1, 2, or 3, in accordancewith various aspects of the techniques described in this disclosure.

FIG. 5 is an example flowchart illustrating an example method forapplying inverse warping to a warped frame, in accordance with variousaspects of the techniques described in this disclosure.

FIG. 6 is an example flowchart illustrating an example methods andtechniques for tracking indicated area(s) to subsequent frames andwarping subsequent frames based on the tracked area(s), in accordancewith various aspects of the techniques described in this disclosure.

FIG. 7 is an example illustration depicting at least a portion of aframe, an area indicated as an area of importance, and an unwarped frameof example image data, in accordance with various aspects of thetechniques described in this disclosure.

FIG. 8 is an example illustration depicting at least a portion of aframe and a grid warping around an indicated area of importance in orderto selectively allocate processing resources to the indicated area, inaccordance with various aspects of the techniques described in thisdisclosure.

FIG. 9 is an example illustration depicting at least a portion of ascaled frame having various levels of digital scaling applied inaccordance with warping grid areas, in accordance with various aspectsof the techniques described in this disclosure.

FIG. 10 is an example illustration depicting at least a portion of ascaled frame having various levels of digital scaling applied inaccordance with warping grid areas, in accordance with various aspectsof the techniques described in this disclosure.

DETAILED DESCRIPTION

A camera, including camera processor(s) and image sensors, may captureframes of image data using the one or more image sensors and output thecaptured frames, including pixel information defining the frame, to thecamera processor(s) for processing. Among other processing techniquesemployed, the camera processor(s) may perform various scaling techniquesunder various circumstances upon receiving the image data in order toachieve a desired output resolution (e.g., a desired video outputresolution) or to comply with limited processing capabilities.

In one example, a camera processor may perform upscaling using knownpixel values to achieve an output resolution having more pixels than thenumber of pixels available from a particular image sensor. For example,a camera processor may perform upscaling to achieve a desired outputresolution in the case of a relatively small image sensor to achievevideo of a higher resolution, such as a 4K-capable system with a 4Kimage sensor (3840×2160 output pixels/frame) and an 8K desired videooutput (7680×4320 display pixels/frame). In another example, a cameraprocessor may perform downscaling based on the pixel information toachieve an output resolution having fewer pixels compared to the numberof pixels available from a particular image sensor. Moreover, a cameraprocessor may perform downscaling in cases where the camera has limitedprocessing capabilities in terms of resolution, such as a high framerate application (e.g., 240 frames per second (fps)) using an 8K imagesensor that is fully utilized in snapshot or picture mode, but using the8K image sensor for video at high frame rates, where the camera hasvideo processing capabilities of 4K at 120 fps. Downscaling, however,tends to degrade the resulting image or video quality. In either case,camera processors may attempt to uniformly apply upscaling ordownscaling to all pixels of an entire frame at a constant level ofdigital scaling regardless of any areas or regions of the frameindicated as having a higher degree of importance in the frame relativeto areas adjacent such indicated areas. It may be advantageous, however,for a camera processor to process areas indicated as having a higherdegree of importance differently than areas adjacent such indicatedareas.

The aforementioned problems, among others, may be addressed by thedisclosed techniques for providing dynamic resolution allocation byallocating more area of a frame to one or more user selected,automatically detected, or otherwise indicated areas of the framecompared to other areas of the frame adjacent the one or more indicatedareas by warping a frame around the indicated areas or otherwise warpinga frame that includes an indicated area of importance. In some examples,camera processor(s) may perform the warping operation by geometricallydisplacing various pixels in the frame. Specifically, cameraprocessor(s) may enlarge particular areas of the frame, such that pixelsin the indicated area undergo geometric displacement, whereas pixels inadjacent areas may be displaced, such that the adjacent areas decreasein size. As such, the camera processor(s) may apply scaling levels inamounts proportional to the amount of area of the frame allocated theone or more indicated areas through application of a warping grid. Inany event, the warping is applied around indicated areas of the frame,including user-indicated areas of importance and/or automaticallydetermined areas of importance, such that more area of a frame isallocated to the one or more indicated areas. In some instances, a videoencoder/decoder or camera processor, such as an image processing engine,may process pixels of the warped frame, which in some instances mayfurther result in unwarping of the warped frame.

In an illustrative example, when upscaling to achieve a higher desiredoutput resolution (e.g., 8K video), the camera processor(s) may upscalethe indicated area(s) more compared to areas adjacent the indicatedarea(s) due to the warped frame allocating more pixels to the indicatedarea(s). Selective allocation of processing resources in this context isadvantageous, for example, because the camera processor(s) may then beable to perform upscaling processing of various indicated areas of aframe before the frame undergoes post-processing, such as temporalprocessing. This, in turn, reduces the amount of scaling used duringsuch post-processing processes. In addition, upscaling a portion ofimage data before performing certain processing techniques, such astemporal super-resolution, sharpening, etc., may improve image or videoquality in that portion of image data. Similarly, camera processor(s)may perform more temporal processing (e.g., temporal super-resolution),denoising, sharpening, etc. on pixels relating to the indicated areasbecause the indicated areas are allocated more area (e.g., more pixels)of the warped frame. That is, camera processor(s) or encoder/decodersmay process the indicated areas more than other areas upon receiving thewarped frame.

In some examples, selective allocation of processing resources may beachieved by using dynamic object-based lens distortion applied to aframe of image data including an area indicated by user selection orautomatic detection before post-processing. The indicated area may beupscaled, or in some instances not scaled, while areas adjacent theindicated area may be downscaled. In this way, the areas adjacent theindicated area may be scaled (e.g., sampled) less densely (e.g., withfewer pixels) compared to the indicated area. In addition, the cameraprocessor(s) workload may be maintained at an overall resolution (e.g.,1080p). Following processing via camera processor(s), areas adjacent theindicated area may be upsampled to correct for the warping operation.For example, the camera processor(s) may upscale (e.g., upsample) theareas adjacent the indicated area to match the resolution of theindicated area but using fewer pixels. As such, the disclosed technologymay find advantages in the context of high frame rate processes whereresolution may be limited.

As is described in detail below, the camera processor(s) may warp aframe around an indicated area to create a warped frame. The cameraprocessor(s) may then process the warped frame of image data accordingto warped proportions of the warped frame, where more area of the frameis allocated to one or more indicated areas compared to an amount ofarea of the frame allocated the one or more indicated areas prior toapplication of the warping grid. In other instances, the cameraprocessor(s) may output an unwarped frame at a desired output resolutionthat is scaled using an inverse distortion operation that applies aninverse warping grid to remove warping applied by the warping grid. Thecamera processor(s), or in some cases, a video encoder, may apply theinverse distortion operation on the warped frame to create an unwarpedframe having a higher output resolution relative to the input resolutionof the warping operation.

FIG. 1 is a block diagram of a computing device 10 configured to performone or more of the example techniques described in this disclosure.Examples of computing device 10 include a computer (e.g., personalcomputer, a desktop computer, or a laptop computer), a mobile devicesuch as a tablet computer, a wireless communication device (such as,e.g., a mobile telephone, a cellular telephone, a satellite telephone,and/or a mobile telephone handset), an Internet telephone, a digitalcamera, a digital video recorder, a handheld device, such as a portablevideo game device or a personal digital assistant (PDA), a drone device,or any device that may include one or more cameras.

As illustrated in the example of FIG. 1, computing device 10 includesone or more image sensor(s) 12. Image sensor(s) 12 may be referred to insome instances herein simply as “sensor 12,” while in other instancesmay be referred to as a plurality of “sensor(s) 12” where appropriate.In some examples, sensor(s) 12 of FIG. 1 may represent one or more imagesensor(s) that may include an array of pixel sensors (e.g., “pixels”).In addition, sensor(s) 12 may include processing circuitry, memory,and/or combinations thereof. Computing device 10 further includes one ormore camera processor(s) 14. In some instances, camera processor(s) 14may comprise an image signal processor (ISP) that uses variousprocessing algorithms under various circumstances to process image data.As shown in FIG. 1, a camera 15 may refer to a collective deviceincluding one or more image sensor(s) 12 and one or more cameraprocessor(s) 14. Multiple cameras 15 may be included with a singlecomputing device 10, such as a mobile phone having one or more frontfacing cameras and/or one or more rear facing cameras.

In accordance with various techniques of this disclosure, cameraprocessor(s) 14 may receive one or more frames of image data fromsensor(s) 12. That is, camera processor(s) 14 are configured to receiveimage frames (e.g., pixel data) from sensor(s) 12, and process the imageframes to generate image and/or video content. For example, sensor(s) 12may be configured to capture individual frames, frame bursts, framesequences for generating video content, photo stills captured whilerecording video, image previews, or motion photos from before and/orafter capture of a still photograph. CPU 16, GPU 18, camera processor(s)14, or some other circuitry may be configured to process the imageand/or video content captured by sensor(s) 12 into images or video fordisplay on display 28. In the context of this disclosure, image framesmay generally refer to frames of data for a still image or frames ofvideo data or combinations thereof, such as with motion photos. Cameraprocessor(s) 14 may receive from sensor(s) 12 pixel data of the imageframes in any format. For example, the pixel data may include differentcolor formats, such as RGB, YCbCr, YUV, etc.

While some example techniques are described herein with respect to asingle sensor 12, the example techniques are not so limited, and may beapplicable to various camera types used for capturing images/videos,including devices that include multiple image sensors, multiple lenstypes, and/or multiple camera processors. For example, computing device10 may include dual lens devices, triple lens devices, etc. In someexamples, one image sensor 12 may be allocated for each lens. That is,multiple image sensors 12 may be each allocated to different lens types(e.g., wide lens, ultra-wide lens, telephoto lens, and/or periscopelens, etc.). In some examples, a single image sensor 12 may correspondto multiple lenses.

In some examples, a single one of camera processor(s) 14 may beallocated to one or more sensors 12. In some instances, however,multiple camera processors 14, such as multiple image front ends (IFEs)or multiple image processing engines (IPEs), may be allocated to one ormore sensor(s) 12. In general, an IFE performs real-time processing ofimage data received from image sensor(s) 12. An IFE also provides aninterface between image sensor(s) 12 and one or more IPEs. For example,the IFE may pre-process image data, such as by warping a frame of imagedata around an area of importance as discussed in this disclosure, andoutput the warped frame to the IPE. Generally speaking, an IPE performsvarious offline processing techniques, including denoising, spatialdenoising, temporal denoising, edge enhancement, sharpening, scaling(e.g., upscaling, downscaling), color correction, etc.

In examples including multiple camera processor(s) 14, cameraprocessor(s) 14 may share sensor(s) 12, where each of cameraprocessor(s) 14 may interface with each of sensor(s) 12. In any event,camera processor(s) 14 may initiate capture of a video or image of ascene using a plurality of pixel sensors of sensor(s) 12. In someexamples, a video may include a sequence of individual frames. As such,camera processor(s) 14 causes sensor(s) 12 to capture the image usingthe plurality of pixel sensors. Sensor(s) 12 may then output pixelinformation to camera processor(s) 14 (e.g., pixel values, luma values,color values, charge values, Analog-to-Digital Units (ADU) values,etc.), the pixel information representing the captured image or sequenceof captured images. In some examples, camera processor(s) 14 may processmonochrome and/or color images to obtain an enhanced color image of ascene.

As illustrated, computing device 10 may further include a centralprocessing unit (CPU) 16, an encoder/decoder 17, a graphics processingunit (GPU) 18, local memory 20 of GPU 18, user interface 22, memorycontroller 24 that provides access to system memory 30, and displayinterface 26 that outputs signals that cause graphical data to bedisplayed on display 28 (e.g., a display device).

CPU 16 may comprise a general-purpose or a special-purpose processorthat controls operation of computing device 10. A user may provide inputto computing device 10, for example, to cause CPU 16 to execute one ormore software applications. The software applications that execute onCPU 16 may include, for example, a camera application, a graphicsediting application, a media player application, a video gameapplication, a graphical user interface application or another program.For example, a camera application may allow the user to control varioussettings of camera 15. The user may provide input to computing device 10via one or more input devices (not shown) such as a keyboard, a mouse, amicrophone, a touch pad or another input device that is coupled tocomputing device 10 via user interface 22. For example, user interface22 may receive input from the user to select a particular area (e.g., anarea of importance) of a frame, adjust a desired digital zoom levels,alter aspect ratios of image data, record video, take a snapshot whilerecording video, apply filters to the image capture, record slow motionvideo or super slow motion video, apply night shot settings, capturepanoramic image data, etc.

One example of the software application is a camera application. CPU 16executes the camera application, and in response, the camera applicationcauses CPU 16 to generate content that display 28 outputs. For instance,display 28 may output information such as light intensity, whether flashis enabled, and other such information. The user of computing device 10may interface with display 28 to configure the manner in which theimages are generated (e.g., with or without flash, focus settings,exposure settings, and other parameters). The camera application alsocauses CPU 16 to instruct camera processor(s) 14 to process the imagescaptured by sensor(s) 12 in the user-defined manner. For example, CPU 16may receive via user interface 22 indication of a user-selected area ofa frame (e.g., a face).

In some examples, CPU 16 may receive user input with respect to an areaof a frame indicating an area of importance. The user input may includetap, gaze, gesture, and/or voice inputs. For example, CPU 16 may track agaze of a user to identify particular areas of a frame that include aface (e.g., face detection, facial recognition applications, etc.). CPU16 may then track the face, indicated by the user input, across multipleframes. As such, the indicated area of importance may be anobject-of-interest, a region-of-interest, etc. In another example, thearea of importance may include a person, or at least a portion of aperson, selected in a picture or video frame (e.g., user tap, user voiceinput). In another example, the area of importance may include a regionof a frame, such as a center quarter of the frame, top-right region ofthe frame, etc. Example gesture inputs may include pinch-to-zoomcommands, screen swipes, detected hand gesture commands, etc. In someexamples, the indicated area may include an automatically detected hand,head, face, or other body gesture, such as in the case of hand gesturerecognition or other forms of gesture recognition. The gesture may thenbe tracked so as to provide user input or additional user input. Anexample voice input may include a user verbalizing a particular objector region in a frame that the user would like CPU 16 or cameraprocessor(s) 14 to identify and/or track. In any event, the indicatedarea may include any area of a frame that a user desires to capture withmore resolution than another area of the frame. CPU 16 may executetracking of the indicated area from a first frame to a subsequent frame.In some examples, camera processor(s) 14 may perform detecting and/ortracking of the one or more indicated areas. For example, cameraprocessor(s) 14 may use machine learning (ML) and/or artificialintelligence (AI) algorithms to detect and/or track variously indicatedareas of a frame. In some examples, the user input may be received viauser interface 22, where CPU 16 and/or camera processor(s) 14 may thenperform area identification and/or tracking of areas (e.g., areas ofimportance) of a frame of image data variously indicated via userinterface 22.

In some examples, CPU 16 may deploy, as part of a software application,ML models and/or AI algorithms to detect areas of importance of a frame.CPU 16 may indicate the detected areas or portions of the areas ofimportance (e.g., portions of a face), to camera processor 14 that theframe of image data should be warped so as to allocate more area of theframe (e.g., more pixels) to the indicated area. In some instances, CPU16 may deploy such algorithms or models in conjunction with user input.In some examples, CPU 16 may deploy such algorithms or modelsindependent of user input. As such, the ML models and/or AI algorithmsmay execute on CPU 16. In another example, the ML models and/or AIalgorithms may execute on GPU 18. In another example, certain ML modelsand/or AI algorithms tasked with indicating areas of a frame forprocessing may execute on hardware accelerators dedicated for ML/AItasks. For example, a DSP may execute certain ML models and/or AIalgorithms configured to indicate particular areas of a frame. In someexamples, an object detection network, such as a deep learning-baseddetector, may be used to detect or classify object in frames of imagedata. For example, object detection architecture (e.g.,you-only-look-once (YOLO), single shot detector (SSD), convolutionalneural network (CNNs), etc.) may be used to indicate locations ofcertain areas (e.g., locations of faces) in a frame. In another example,CPU 16 may utilize a network-based face detection to indicate variousareas or regions of a frame as containing areas of importance.

In some examples, an AI algorithm may be configured to automaticallyrecognize or detect particular areas in a frame. As such, an AIalgorithm and/or a ML model may automatically detect an area of a frameand track the area in the frame or over a sequence of frames, such as apreview frame prior to capturing a photo, a sequence of frames forvideo, etc., regardless of whether user input has been received. In someexamples, CPU 16 may use AI algorithms and/or ML models to detect anarea of importance after receiving user input. For example, a user maytouch part of a display of the captured image that corresponds to aparticular object, in which case an AI algorithm may detect theparticular object in the frame so as to indicate that the full objectcomprises an area of importance. In some examples, an AI algorithm mayautomatically detect an area of importance. In addition, the AIalgorithm may detect a user-selected area of importance, such as theface of a person or hand gesture of a person (e.g., for facial and humangesture recognition applications). The AI algorithm may also detectother user input, such as a gaze, gesture (e.g., hand gesture), voice,or tap input. In some examples, camera processor(s) 14 may performdetecting and/or tracking of such variously indicated areas. Forexample, camera processor(s) 14 may detect and/or track areas indicatedwithin a frame using an AI algorithm or ML model to automatically detectan area of importance and in addition, detect a user-selected area ofimportance.

In accordance with various techniques of this disclosure, cameraprocessor(s) 14 may receive an indication of an area of importancecorresponding to at least a portion of a frame received from sensor(s)12. Camera processor(s) 14 may receive an indication of the area fromCPU 16 or in some instances, camera processor(s) 14 may receive theindication of the area with or without user input as a result of aprocess implemented by camera processor(s) 14.

Camera processor(s) 14 may warp the frame around an indicated area ofimportance to create a warped frame having the indicated area enlargedin the warped frame. In some examples, camera processor(s) 14 may applya warping grid around the indicated area to create the warped frame.That is, a warping operation may be performed that warps a framerelative to the location of the indicated area in the frame, such thatthe indicated area appears enlarged, whereas other areas adjacent theindicated area would appear shrunken in size. In an example, cameraprocessor(s) 14 may enlarge an area of importance to have more pixelsand shrink areas adjacent the indicated area to include fewer pixels inorder to create the warped frame, such that the total area of the frameremains the same before and after the warping operation is complete.

Camera processor(s) 14 may then output the warped frame for processing.In some instances, applying the warping grid to create the warped frameincludes applying the warping grid to the entire frame including aroundthe indicated area(s) of importance. For example, applying the warpinggrid to create the warped frame includes applying the warping grid tothe frame including to the indicated area of importance and at least onearea of the frame adjacent the indicated area. In this way, applicationof the warping grid may result in more area of the warped frame (e.g.,more pixels) being allocated to the indicated area relative to an amountof area of the warped frame allocated to at least one area of the frameadjacent the indicated area of importance.

Camera processor(s) 14 may output the warped frame for furtherprocessing. For example, as described with reference to FIGS. 2 and 3,camera processor(s) 14 may output the warped frame for scalingoperations to achieve a desired output resolution. In addition, cameraprocessor(s) 14, such as IPE 42A and/or IPE 42B, may perform additionalprocessing operations to the warped frame in accordance with the warpingof the frame. For example, camera processor(s) 14, IPE 42A and/or IPE42B, may perform spatial denoising, temporal denoising, edgeenhancement, sharpening, scaling (e.g., upscaling, downscaling), colorcorrection, etc.

In some examples, scaling operations may be performed by encoder/decoder17, or by other processors of camera processor(s) 14, such as by one ormore IPEs. In some examples, camera processor(s) 14 may output thewarped frame to system memory 30. In another example, cameraprocessor(s) 14 or encoder/decoder 17 may perform various scalingoperations to unwarp the warped frame and as such, store the unwarpedframe to system memory 30. In some examples, unwarping of the frameinvolves an inverse distortion operation. In any event, cameraprocessor(s) 14 or encoder/decoder 17 may unwarp a frame warped aroundan indicated area by causing the previously enlarged area to return toan original size of the area of importance relative to other areas ofthe frame as in the original unwarped frame received from imagesensor(s) 12. In such examples, camera processor(s) 14 may upscale areasadjacent the previously indicated areas at various scaling levels, whilethe indicated area may undergo scaling at different levels (e.g., anon-scaling level, etc.), to achieve the unwarping operation. Thescaling levels may be proportional to the amount of pixels allocatedvarious areas of the frame during the original warping operation. Forexample, the indicated areas may receive more pixels as a result of ageometric displacement of pixels during the frame warping operation.Camera processor(s) 14 or encoder/decoder 17 may perform unwarping afteror during offline processing.

Memory controller 24 facilitates the transfer of data going into and outof system memory 30. For example, memory controller 24 may receivememory read and write commands, and service such commands with respectto memory 30 in order to provide memory services for the components incomputing device 10. Memory controller 24 is communicatively coupled tosystem memory 30. Although memory controller 24 is illustrated in theexample of computing device 10 of FIG. 1 as being a processing circuitthat is separate from both CPU 16 and system memory 30, in otherexamples, some or all of the functionality of memory controller 24 maybe implemented on one or both of CPU 16 and system memory 30.

System memory 30 may store program modules and/or instructions and/ordata that are accessible by camera processor(s) 14, CPU 16, and GPU 18.For example, system memory 30 may store user applications (e.g.,instructions for the camera application), resulting images or framesfrom camera processor(s) 14 and/or encoder/decoders 17, etc. Systemmemory 30 may additionally store information for use by and/or generatedby other components of computing device 10. For example, system memory30 may act as a device memory for camera processor(s) 14. System memory30 may include one or more volatile or non-volatile memories or storagedevices, such as, for example, random access memory (RAM), static RAM(SRAM), dynamic RAM (DRAM), read-only memory (ROM), erasableprogrammable ROM (EPROM), electrically erasable programmable ROM(EEPROM), flash memory, a magnetic data media or an optical storagemedia.

In some examples, system memory 30 may include instructions that causecamera processor(s) 14, CPU 16, GPU 18, and display interface 26 toperform the functions ascribed to these components in this disclosure.Accordingly, system memory 30 may be a computer-readable storage mediumhaving instructions stored thereon that, when executed, cause one ormore processors (e.g., camera processor(s) 14, CPU 16, GPU 18, anddisplay interface 26) to perform various functions.

In some examples, system memory 30 is a non-transitory storage medium.The term “non-transitory” indicates that the storage medium is notembodied in a carrier wave or a propagated signal. However, the term“non-transitory” should not be interpreted to mean that system memory 30is non-movable or that its contents are static. As one example, systemmemory 30 may be removed from computing device 10, and moved to anotherdevice. As another example, memory, substantially similar to systemmemory 30, may be inserted into computing device 10. In certainexamples, a non-transitory storage medium may store data that can, overtime, change (e.g., in RAM).

Camera processor(s) 14, CPU 16, and GPU 18 may store image data, and thelike, in respective buffers that are allocated within system memory 30.Display interface 26 may retrieve the data from system memory 30 andconfigure display 28 to display the image represented by the generatedimage data. In some examples, display interface 26 may include adigital-to-analog converter (DAC) that is configured to convert thedigital values retrieved from system memory 30 into an analog signalconsumable by display 28. In other examples, display interface 26 maypass the digital values directly to display 28 for processing.

In addition, camera processor(s) 14 may be configured to analyze pixeldata and/or output the resulting images (e.g., pixel values for each ofthe image pixels) to system memory 30 via memory controller 24. Each ofthe images may be further processed for generating a final image fordisplay. For example, GPU 18 or some other processing unit, includingcamera processor(s) 14, may perform color correction, white balance,blending, compositing, rotation, or other operations to generate thefinal image content for display.

Display 28 may include a monitor, a television, a projection device, aliquid crystal display (LCD), a plasma display panel, a light emittingdiode (LED) array, an organic LED (OLED), a cathode ray tube (CRT)display, electronic paper, a surface-conduction electron-emitted display(SED), a laser television display, a nanocrystal display or another typeof display unit. Display 28 may be integrated within computing device10. For instance, display 28 may be a screen of a mobile telephonehandset, a tablet computer, or a laptop. Alternatively, display 28 maybe a stand-alone device coupled to computing device 10 via a wired orwireless communications link. For instance, display 28 may be a computermonitor or flat panel display connected to a personal computer via acable or wireless link.

Also, although the various components are illustrated as separatecomponents, in some examples the components may be combined to form asystem on chip (SoC). As an example, camera processor(s) 14, CPU 16, GPU18, and display interface 26 may be formed on a common integratedcircuit (IC) chip. In some examples, one or more of camera processor(s)14, CPU 16, GPU 18, and display interface 26 may be in separate ICchips. Various other permutations and combinations are possible, and thetechniques of this disclosure should not be considered limited to theexample illustrated in FIG. 1.

The various components illustrated in FIG. 1 (whether formed on onedevice or different devices), including sensor(s) 12 and cameraprocessor(s) 14, may be formed as at least one of fixed-function orprogrammable circuitry, or a combination of both, such as in one or moremicroprocessors, application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs), digital signal processors (DSPs), orother equivalent integrated or discrete logic circuitry. Examples oflocal memory 20 include one or more volatile or non-volatile memories orstorage devices, such as random-access memory (RAM), static RAM (SRAM),dynamic RAM (DRAM), erasable programmable ROM (EPROM), electricallyerasable programmable ROM (EEPROM), flash memory, a magnetic data mediaor an optical storage media.

The various structures illustrated in FIG. 1 may be configured tocommunicate with each other using bus 32. Bus 32 may be any of a varietyof bus structures, such as a third-generation bus (e.g., aHyperTransport bus or an InfiniBand bus), a second-generation bus (e.g.,an Advanced Graphics Port bus, a Peripheral Component Interconnect (PCI)Express bus, or an Advanced eXtensible Interface (AXI) bus) or anothertype of bus or device interconnect. It should be noted that the specificconfiguration of buses and communication interfaces between thedifferent components shown in FIG. 1 is merely exemplary, and otherconfigurations of computing devices and/or other image processingsystems with the same or different components may be used to implementthe techniques of this disclosure.

FIG. 2 is a block diagram illustrating example components of camera 15including camera processor(s) 14 of FIG. 1. In some examples, cameraprocessor(s) 14 may include an IFE 40. In some examples, IFE 40interfaces between image sensor(s) 12 and one or more IPEs 42. IPEs 42perform offline processing of image data as described below. In someexamples, one of IPEs 42 may perform real-time processing of image data.That is, IPEs 42 may be configured to perform one or more of theprocessing techniques offline, rather than in real-time as the imagedata is received from sensor(s) 12. In some examples, one or more IPEs42 may be configured to perform techniques such as, spatial processing,temporal processing, sharpening, denoising, etc. In addition, cameraprocessor(s) 14 may be configured to perform various operations on imagedata captured by sensor(s) 12, including auto white balance, colorcorrection, or other post-processing operations. In some examples, aprocessing “engine” may refer to hardware that can be used to processdata. Example hardware may include fixed-function or programmableprocessing engines in camera processor(s) 14, GPU 18, CPU 16, a DSP, orany other processing circuitry available on computing device 10.

In some examples, CPU 16 may include an area identification system 34that identifies one or more areas of a frame (AOFs), such asregions-of-interest (ROIs), areas of importance, objects of importance,etc., found within a particular frame of image data (hereinafter, AOFidentification system 34). AOF system 34 may identify the one or moreAOFs with or without user input. In some examples, AOF identificationsystem 34 may identify an AOF based on user input that includes one ormore of a tap, gaze, gesture, voice, etc. In an illustrative example,AOF identification system 34 may track the gaze of a user. In anotherexample, AOF identification system 34 may receive a touch input, such asa tap, indicating the location of a face for identifying and/or trackingin a frame of image data. It should be noted that while certain examplesof user input, such as touch or tap input, may be described in thisdisclosure, the techniques of this disclosure are not so limited, andother examples of user input may be used to indicate areas of a framefor identification and/or tracking. In some examples, AOF identificationsystem 34 may identify the AOF using AI algorithms and/or ML models. Inaddition, AOF identification system 34 may perform automatic detectionof one or more AOFs as an alternative to user input or to supplement theuser input.

Similarly, CPU 16 may include an AOF tracking system 36 that tracks anAOF, or portions of an AOF from one frame to a subsequent frame. AOFidentification system 34 or AOF tracking system 36 may utilize variousML models or AI algorithms to identify an AOF and/or track the AOFacross multiple frames. In some examples, either AOF identificationsystem 34 or AOF tracking system 36 may perform image segmentation. In anon-limiting example, AOF identification system 34 or AOF trackingsystem 36 may perform image segmentation based on color. For example,AOF identification system 34 or AOF tracking system 36 may perform imagesegmentation based on the color(s) in a region of a frame selected by auser. In some examples, AOF identification system 34 or AOF trackingsystem 36 may identify an area of a frame and/or track the area of theframe using various feature detection and/or optical flow techniques.

IFE 40 may also include a frame warping processor 44A configured to warpa frame of image data received from image sensor(s) 12. In someexamples, frame warping processor 44A may apply virtual lens distortionor a warping grid to one or more frames. Frame warping processor 44A mayreceive from CPU 16 an indication of an area of the frame, such asselected or detected area of importance. For example, the indication ofan area of importance may include size, shape, and/or frame locationinformation. In one example, the area may indicate an area of the framewhere an object of importance is located, the size of the object, andthe shape of the object, such that frame warping processor 44A may warpthe frame around the area of importance based on the informationreceived from CPU 16.

In some examples, IFE 40 may warp a frame of image data via framewarping processor 44A. For example, IFE 40 may warp the frame around anarea of the frame indicated as including an area of importance. In someexamples, IFE 40 may warp the frame around an area of importance as theframe of image data is received from image sensor(s) 12. In anon-limiting example, IFE 40 may apply a warping grid to a frame as theframe of image data is received from image sensor(s) 12. As such, theIFE 40 may create a warped frame that allocates different size areas ofthe warped frame to variously indicated areas compared to areas adjacentthe indicated areas. IFE 40 may warp the frame around the indicated areawhile maintaining a constant number of pixels for the frame. As such,warping the frame allocates more pixels to the indicated area in thewarped frame while allocating fewer pixels to areas of the frameadjacent to the indicated area. However, the warped frame may have thesame number of pixels as the frame received from image sensor(s) 12. Insome examples, one or more of IPEs 42 may include a frame warpingprocessor, such as frame warping processor 44B. The IPE 42 may includeframe warping processor 44B in addition to or in lieu of IFE 40including frame warping processor 44A. In such instances, IPE 42 mayreceive a frame of image data from IFE 40 and may apply a warpingoperation to the frame using frame warping processor 44B. After warping,IPE(s) 42 may apply various processing operations to the warped frame,while in some examples, IPE(s) 42 may apply such operationsnon-uniformly to the warped frame based on the particular warpingconfiguration of the output warped frame.

In some examples, after warping, IPE(s) 42 may apply various processingresources to the frame, such as spatial denoising, temporal denoising,temporal and/or spatial super-resolution, tone mapping, warping (e.g.,geometrical warping), edge enhancement, sharpening, scaling (e.g.,upscaling, downscaling), color correction, etc. In one example, afterwarping, IPE(s) 42 may perform scaling of the warped frame usingdifferent scaling levels with respect to pixels corresponding to theindicated AOFs (e.g., areas of importance, etc.) compared to areasadjacent the indicated areas. In another example, IPE(s) 42 may performtemporal denoising by combining frames of video data. While someexamples of this disclosure include the example of scaling usingdifferent scaling levels across a warped frame, the techniques of thisdisclosure are not so limited, and it will be understood that IPE(s) 42or encoder/decoder 17 may allocate more processing resources, such asthose described above, to indicated AOFs relative to other areas in aframe adjacent the indicated AOFs.

In an illustrative example, after warping, IPE(s) 42 may perform scalingof the warped frame using different scaling levels. As such, IPE(s) 42or encoder/decoder 17 may allocate different scaling levels to indicatedareas (e.g., areas of importance) relative to other areas in a frame,such that the indicated areas receive enhanced resolution. In any event,by allocating an indicated area of importance more pixels, IPE(s) 42 orencoder/decoder 17 may allocate more time and resources on the indicatedarea in the warped frame relative to the indicated area in an unwarpedframe of the image data.

In some examples, IPE(s) 42 may include an inverse frame warpingprocessor 46 that is configured to unwarp a warped frame of image data.As discussed with reference to FIG. 3, inverse frame warping processor46 may perform unwarping after or during offline processing. In anotherexample, encoder/decoder 17 may perform the unwarping as encoder/decoder17 receives data from camera processor(s) 14.

In this way, better processing, quality, resolution, etc. is achievedfor indicated areas (e.g., areas of importance) relative to areasadjacent the indicated areas. In addition, in implementations allowinghigh processing capabilities, selectively allocating processingresources based on parameters of a warped frame may also reduce powerconsumption of computing device 10. That is, camera processor(s) 14 maynot apply processing resources equally to important and non-importantareas of the warped frame, although the device may be configured toperform such high resolution processing. In this way, cameraprocessor(s) 14 may increase efficiency and conserve energy that mayotherwise be wasted for resource-intensive processing of non-importantareas of a frame.

FIG. 3 is a conceptual diagram illustrating example real-time processingand offline processing of camera processor(s) 14. Real-time processing200 may be a type of live stream. A live stream refers to anyconfiguration in which the processing engines receive data from imagesensor(s) 12, but do not modify data from previous requests (e.g., datapreviously stored in memory). Offline processing 220 may be a type ofoffline stream. An offline stream is any configuration in which theprocessing engines do not receive data from an image sensor, but ratheraccess image data from a memory or from another processing engine.

3A 206 represents the functionality of a statistics algorithm processingengine of camera processor(s) 14. The 3A algorithms may includeautofocus (AF), auto exposure control (AEC), and auto white balance(AWB). 3A 206 may be configured to perform one or more of AF, AEC,and/or AWB processing to control the function of image sensor(s) 12.IPEs 42A-42B (collectively, IPEs 42) represent image processing enginesinternal to camera processor(s) 14. IPEs 42A may be configured to handlevarious real-time image processing techniques (e.g., image processingthat occurs at the same speed as image sensor(s) 12 throughput rate). Insome examples, IPEs 42 may be configured to perform processingtechniques for video and image preview, such as 3A processing via 3A206. System memory 30 represents memory (e.g., frame buffers of systemmemory 30) used for displaying image data for camera and camcorder(video) application. Camera processor(s) 14 may perform 3A algorithmsbased on the warped frame outputted from IFE 40.

Offline processing 220 is configured to process image data, afterprocessing by IFE 40. In this way, offline processing 220 may beconsidered a post-processing. Unlike real-time processing 200, offlineprocessing 220 does not access image data directly from camera module 12(e.g., sensor(s) 12), but rather accesses data from IFE 40, eitherdirectly or through an output buffer of IFE 40. In addition, offlineprocessing 220 may include image processing that does not need to occurat the same speed as image sensor(s) 12 throughput rate. Offlineprocessing 220 may include IPE 42B. System memory 30 may represent thememory to which the processed image data is to be stored. System memory30 may further represent the memory to which the pre-processed imagedata is to be stored (e.g., a warped frame). In such examples, systemmemory 30 may perform unwarping and scaling operations of the warpedframe or may output the warped frame to another device for furtherprocessing.

In an illustrative example, IFE 40 may receive one or more frames ofimage data from image sensor(s) 12, the one or more frames including atleast one area of importance. The area of importance information may bereceived from CPU 16, such as from AOF identification system 34. In someexamples, AOF identification system 34 may indicate to CPU 16 an AOFthat constitutes an area of the frame selected by a user and/orautomatically detected. For example, IFE 40 may receive one or moreframes of image data from image sensor(s) 12 with image segmentationinformation regarding the location, size, and shape of an area indicatedin at least one frame of the image data.

IFE 40 may provide at least one frame of the image data to IPE 42A orIPE 42B. As such, IPE 42A or IPE 42B may warp a frame of image dataaround the indicated area of importance to create the warped frame. Forexample, IPEs 42 may warp the frame of image data in real-time as IPEs42 read input from IFE 40 or offline as part of an offline processingpipeline 220. In some examples, IFE 40 may apply the warping grid aroundthe area of importance to create the warped frame. For example, IPEs 42may apply a warping grid in real-time as IPEs 42 read input from IFE 40or offline as part of an offline processing pipeline 220. In any event,IPE 42A or IPE 42B may output the warped frame to system memory 30 orencoder/decoder 17.

In some examples, IPEs 42A or 42B may perform scaling of the warpedframe to achieve an unwarped frame at a desired output resolution. Insome examples, IPEs 42A or 42B may perform scaling prior to outputtingthe frame of image data to system memory 30 or encoder/decoder 17. Insome examples, encoder/decoder 17 may perform an inverse distortionoperation that applies inverse warping to remove warping of the warpedframe. For example, encoder/decoder 17 may perform an inverse distortionoperation that applies an inverse warping to remove warping of thewarped frame. For example, encoder/decoder 17 may perform an inversedistortion operation that applies an inverse warping grid to removewarping applied by a warping grid. Encoder/decoder 17 may perform theinverse distortion operation in real-time as encoder/decoder 17 readsinput from IPEs 42A or 42B. In some examples, encoder/decoder 17 mayperform the inverse distortion operation as encoder/decoder 17 readsinput from system memory 30. For example, encoder/decoder 17 may accessa warped frame of image data from system memory 30. In any event, cameraprocessor(s) 14 may output the frame of image data at the desired outputresolution to one or more of encoder/decoder 17 or a memory device(e.g., DRAM). For example, camera processor(s) 14 may output the frameof image data at the desired output resolution after performing ascaling operation.

In an illustrative example, image sensor(s) 12 may capture a frame ofimage data and output the frame of image data to IFE 40. In someexamples, IFE 40 or IPE 42 may apply a warping operation to the frame ofimage data to create a warped frame of the image data. IFE 40 or IPE 42may output the warped frame. For example, IFE 40 may output the warpedframe to IPE 42A or IPE 42B or in some instances, to another componentof computing device 10, such as a node for 3A processing 206, systemmemory 30, encoder/decoder 17, etc. In another example, IPEs 42A or 42Bmay output the warped frame to another one of IPE 42. In an illustrativeexample, IFE 40 may first output a warped frame to IPE 42A, at whichpoint IPE 42A may output the warped frame to IPE 42B. In some examples,IFE 40, IPE 42A, IPE 42B may create the warped frame and output thewarped frame for further processing, such as scaling and inversedistortion operations.

FIG. 4 is an example flow diagram illustrating example operations ofcamera processor(s) 14, image sensor(s) 12, encoder/decoder 17, and/orsystem memory 30, shown in FIG. 1, performing warping operations tocreate and/or output a warped frame of image data. Camera processor(s)14 may receive a frame of image data from image sensor(s) 12 (402). Insome examples, receiving the frame of image data may comprise IFE 40receiving the frame of image data from image sensor(s) 12.

In addition, camera processor(s) 14 may receive indication of a firstarea (e.g., an area of importance) of a frame (404). As described above,camera processor(s) 14 may receive the indication via one or more of auser selection or an automatic detection. The indicated first area maycomprise at least one of an object-of-interest, such as a face, or aregion-of-interest, such as a particular region that a user indicates asbeing important in the frame. In a non-limiting example, an area ofimportance may represent a portion of a face of a person or a full faceof a person that, in the case of video, may be tracked from one frame toa subsequent frame. In another non-limiting example, the indicated areamay represent one or more objects that move during the filming or photocapture of an activity. In some instances, the higher resolution areafrom a warped frame may be used to assist in the tracking of the area ina subsequent frame. For example, camera processor(s) 14 and/or CPU 16may use the higher pixel areas to determine to what location of theframe a moving object may be tracked in a subsequent frame of the imagedata.

In addition, camera processor(s) 14 may warp the frame around theindicated first area to create a warped frame (406). In one example,camera processor(s) 14 may allocate more pixels to the indicated areaand allocate fewer pixels to areas adjacent the indicated area increating the warped frame. For example, camera processor(s) 14 may usethe indication of the indicated area to determine areas of the framethat should be processed with more processing resources, such as pixelallocation, processing time, etc. In one example, camera processor(s) 14may apply a warping grid around the indicated area. An example visualrepresentation of allocating processing resources non-uniformly across aframe based on variously indicated areas is described with reference toFIG. 8. For example, camera processor(s) 14 may apply a frame distortiontechnique that creates the warped frame, such as a virtual lensdistortion that bends, stretches, or shrinks an image to conform to adistorted virtual lens. The distorted virtual lens may dynamicallydistort images depending on sizes, shapes, and/or locations of areasindicated and/or tracked in a frame of image data. In any event, cameraprocessor(s) 14 may allocate different amount of pixels to differentregions or areas of a frame.

In some examples, camera processor(s) 14 may apply a warping grid aroundthe indicated area to create the warped frame. For example, cameraprocessor(s) 14 may apply the warping grid to the frame including theindicated area and at least one area of the frame adjacent the indicatedarea. In one example, camera processor(s) 14 may apply the warping gridto the frame including to the indicated area, such as an indicated areaof importance, and to at least one area of the frame adjacent theindicated area. In such examples, application of the warping gridresults in more area of the warped frame being allocated to theindicated area relative to an amount of area of the warped frameallocated to at least one area of the frame adjacent the indicated area.

In some examples, camera processor(s) 14 may output the warped frame(408). For example, camera processor(s) 14 may output the warped frameto an encoder/decoder 17, display interface 26 (e.g., for ultimatedisplay via display 28), or system memory 30 (e.g., DRAM). In someexamples, camera processor(s) 14 may output the warped frame from IFE 40to IPE 42A or to IPE 42B. In another example, IPE 42A or IPE 42B mayoutput the warped frame to another one of IPEs 42 (e.g., IPE 42A outputto IPE 42B). In one example, one of IPE(s) 42 may output the warpedframe to system memory 30 for subsequent access by the same one ofIPE(s) 42. For example, one of IPE(s) 42 may perform the frame warpingoperation via frame warping processor 44B and the inverse frame warpingand scaling operation via inverse frame warping processor 46, where theframe warping processor 44B may output the warped frame first to systemmemory 30 or directly to inverse frame warping processor 46. In anotherexample, IFE 40 or one of IPEs 42 may output the warped frame to anothercomponent or logical node of computing device 10, such as 3A node 206,for further processing. In some examples, camera processor(s) 14 mayoutput the warped frame to GPU 18, where GPU 18 may perform an inversewarping operation and/or a scaling operation (e.g., via an inversewarping frame processor) while reading the input of the warped frame orGPU 18 may store the warped frame to local memory 20 for subsequentaccess and processing. In the interest of brevity, not all possiblewarped frame output outcomes may be listed, but it will be understoodthat more output outcomes are possible for the warped frame using cameraprocessor(s) 14 in the context of computing device 10.

FIG. 5 is an example flow diagram illustrating example operations ofcamera processor(s) 14, encoder/decoder 17, and/or system memory 30,shown in FIG. 1, performing inverse warping operations using warpedframes of image data. Encoder/decoder 17 or various components of cameraprocessor(s) 14, such as IPE 42B or IPE 42A, may receive a warped frameof image data from a component applying a warping grid or other warpingtechnique to the frame of image data. For example, IFE 40 may apply thewarping to create the warped frame and output the warped frame to IPE42A, 42B, or encoder/decoder 17. In another example, one of IPEs 42 mayapply the warping to create the warped frame and output the warped frameto encoder/decoder 17.

As such, camera processor(s) 14, CPU 16, GPU 18, display interface 26,or encoder/decoder 17 may perform scaling of the warped frame to achievea desired output resolution (502). In some examples, the scaling mayinclude downscaling, upscaling, no scaling, and combinations thereof, inaccordance with various techniques of this disclosure. That is, thescaling at various levels (e.g., downscaling ratios, upscaling ratios)may not be performed uniformly to the entire warped frame. In someexamples, camera processor(s) 14, CPU 16, GPU 18, display interface 26,or encoder/decoder 17 may perform scaling in accordance with relativepixel amounts that are allocated to various areas of the frame as aresult of the warping operation. In one illustrative example, cameraprocessor(s) 14 may warp a frame of image data to create a warped framethat allocates more area of the frame to indicated areas, includingarea(s) of importance, whereas areas of the frame adjacent the indicatedareas may be allocated less area of the frame (e.g., fewer pixels) as aresult of the warping operation. In such instances, camera processor(s)14, CPU 16, GPU 18, display interface 26, or encoder/decoder 17 mayperform scaling of the warped frame using non-uniform scaling levelsthat vary according to size, shape, and/or location information of theindicated area(s) of the warped frame. As an example, cameraprocessor(s) 14, CPU 16, GPU 18, display interface 26, orencoder/decoder 17 may perform scaling and inverse warping operationswhile receiving and reading in the warped frame as input.

In some examples, camera processor(s) 14 or encoder/decoder 17 mayperform an inverse warping operation by applying inverse warping to thewarped frame (504). Camera processor(s) 14 or encoder/decoder 17 mayapply inverse warping to the warped frame to remove warping from thewarped frame. In one example, camera processor(s) 14 or encoder/decoder17 may apply an inverse warping operation to the warped frame to removewarping from the warped frame. For example, camera processor(s) 14 orencoder/decoder 17 may apply an inverse warping grid that causes theindicated area to return to original proportions of the original frameas a result of the scaling operation. It should be noted that the imageof the indicated area may appear larger in some examples due to certainscaling operations being illustrated in this disclosure as increasing ordecreasing the size of the frame (e.g., FIGS. 9 and 10).

In some examples, IPE 42A, IPE 42B, or encoder/decoder 17 may performscaling of the warped frame to achieve a desired output resolution byperforming an inverse distortion operation. The inverse distortion mayinclude an inverse warping operation to remove warping of the warpedframe. The inversely warped frame may resemble a starting frame prior towarping, except the inversely warped frame may not be of the same sizedue to the scaling (e.g., more or fewer total pixels). For example, toachieve a higher desired output resolution, camera processor(s) 14 orencoder/decoder 17 may perform scaling to the indicated area(s) at ahigher upscaling level or a non-scaling level, relative to areasadjacent the indicated area(s) based on the amount of area of the warpedframe allocated the indicated area(s) of the warped frame.

To achieve a lower desired output resolution or to comply with limitedprocessing capabilities, camera processor(s) 14 or encoder/decoder 17may perform scaling to the indicated areas at a lower downscaling level,such as a non-scaling level, relative to areas adjacent the indicatedareas based on the amount of area of the warped frame allocated theindicated areas. In some instances, camera processor(s) 14 orencoder/decoder 17 may perform upscaling or no scaling for some areas ofthe frame (e.g., variously indicated areas of the frame) and downscalingfor other areas of the frame (e.g., non-indicated areas of the frame).In doing so, camera processor(s) 14 or encoder/decoder 17 may remove thewarping applied by the warping grid and achieve an output frame that isnot warped.

It should also be noted that while inverse warping is describedseparately from scaling, the techniques of this disclosure are not solimited, and it should be understood that scaling and inverse warpingmay be performed in a single operation. For example, camera processor(s)14 may perform a single operation of inverse warping with scaling of thewarped frame to achieve a desired output resolution. The cameraprocessor(s) 14 may perform such operations based at least in part on anidentity map. In another example, camera processor(s) 14 may perform asingle operation of warping and scaling of the warped frame to achievean output resolution that complies with processing limitations thatcamera processor(s) 14, encoder/decoder 17, system memory 30, or display28 may have. That is, camera processor(s) 14 may perform inverse warpingwith scaling of the frame to achieve a smaller frame size, as describedwith reference to FIG. 10 as one illustrative and non-limiting example.In such examples of processing limitations, the smaller frame size mayhave a lower output resolution for the frame as a whole, but where theindicated areas have a higher resolution relative to other areas of theframe (e.g., the adjacent areas of the frame). Similarly, in exampleshaving a higher resolution output, the total frame may have a higherresolution (e.g., more pixels) compared to the frame received from imagesensor(s) 12, with the indicated area having a higher resolutionrelative to other areas of the frame (e.g., the adjacent areas of theframe). For example, the indicated area may have the same or a higherresolution compared to that of the total frame resolution, whereasadjacent areas may have lower resolution compared to that of the totalframe resolution. In some instances, the difference in resolution maynot be noticed by the human eye, such as where more pixels are allocatedto an indicated face of a person and where fewer pixels are allocated toless important background scenery. This is because the human eye is morelikely to focus on the variously indicated areas (e.g., areas ofimportance), rather than the background scenery.

Camera processor(s) 14 or encoder/decoder 17 may output the unwarpedframe of image data. For example, camera processor(s) 14 orencoder/decoder 17 may output the unwarped frame of image data at thedesired output resolution (506). In some examples, camera processor(s)14 or encoder/decoder 17 may output the frame of image data at thedesired output resolution to system memory 30. In another example,camera processor(s) 14 may perform the scaling operation and output theunwarped frame to encoder/decoder 17. In some examples, encoder/decoder17 may perform the scaling operation and encode frames of image data. Insuch instances, encoder/decoder 17 may output the unwarped frames tosystem memory 30 or to an encoding engine.

FIG. 6 is an example flow diagram illustrating example operations ofcamera processor(s) 14, image sensor(s) 12, encoder/decoder 17, and/orsystem memory 30, shown in FIG. 1 performing warping grid operationsusing subsequent frames of image data. Camera processor(s) 14 mayreceive a subsequent frame of image data from image sensor(s) 12 (602).The subsequent frame of image data may include at least a portion of thepreviously indicated area. In an illustrative example, the indicatedarea may include at least a portion of a face indicated by an automaticor semi-automatic facial recognition operation configured to identifyfaces that may appear in a frame of image data and track the one or morefaces across multiple frames of the image data. In such instances, AOFidentification system 34 and/or AOF tracking system 36 may perform thefacial recognition operation.

Camera processor(s) 14 may track the previously indicated first area ofthe frame from a first frame to a subsequent frame (604). In someexamples, the indicated area may be a user-selected area orautomatically detected area in a first frame. In such examples, theindicated area may move relative to the frame parameters, such as withmovement of an object captured in the frame. In another example, theframe may move, such as by movement of camera 15, by a zoom operation,etc. In any event, camera processor(s) 14 may track the indicated areato the subsequent frame. For example, AOF tracking system 36 may trackthe indicated area to the subsequent frame.

In such examples, camera processor(s) 14 may warp the subsequent framearound the tracked area (606). In one example, camera processor(s) 14may apply an updated warping grid around the tracked area or may applyanother warping technique to the subsequent frame so as to indicatevarious scaling levels for various areas of the warped frame. In someexamples, camera processor(s) 14 may be capturing video and may apply awarping grid to each frame of image data captured for the videorecording to create continuously updating warped frames. That is, anupdated warping grid may be a warping grid that warps one or more framesaround indicated areas tracked across frames of the image data. In oneexample, a user may select a face of a person in one frame of video dataas the area of importance, where, for example, the AOF tracking system36 may automatically detect and track the face of the person for theframe and each subsequent frame thereafter. Similar to the output of thefirst warped frame for the first frame of image data, cameraprocessor(s) 14 may output the second warped frame (608). The secondwarped frame may be processed similar to the way in which cameraprocessor(s) 14 or encoder/decoder 17 processes the first warped frameusing scaling and inverse distortion operations.

FIGS. 7 and 8 illustrate example conceptual diagrams of an examplewarping application. In some examples, IFE 40 may perform a framewarping operation by deploying frame warping processor 44A. Framewarping processor 44A may warp a frame of image data received from oneof image sensor(s) 12. In another example, IPE 42 may perform the framewarping operation upon receiving an unwarped frame of image data fromIFE 40. For example, IPE 42 may warp a frame of image data as IPE 42reads the input data received from IFE 40 that includes the frame ofimage data.

While visually depicted as a warping grid in FIGS. 7 and 8, thetechniques of this disclosure are not so limited, and as discussedpreviously, the warping application may involve other warpingtechniques, such as a virtual lens distortion. For sake of conciseness,the illustrative examples of FIGS. 7 and 8 may refer to a warping gridin order to visually illustrate how a frame may be warped around one ormore areas of the frame.

FIG. 7 is an example conceptual illustration 702 of an example portionof a frame 68 of image data. The frame 68 of image data may be capturedby image sensor(s) 12 and output to camera processor(s) 14. In theillustrative example of FIG. 7, frame 68 of image data includes an area70 of frame 68 that has been indicated as having a higher degree ofimportance relative to other areas 74 of frame 68 adjacent indicatedarea 70. In one example, indicated area 70 may include an area of frame68, such as an area of importance, that has been indicated by a user.For example, a user may indicate area 70 using a tap, gaze, gesture, orvoice input. In another example, the indicated area 70 of frame 68 maybe an automatically detected area of frame 68, such as a face or atleast a portion of a face included within frame 68. In some examples,the indicated area 70 may be a user-indicated area that is thenautomatically detected following the user input. The frame of image datamay include areas of the frame defined by a conceptual grid 66 thatpartitions frame 68 into various compartments (e.g., areas 74). Thecompartments may be of uniform area throughout the frame and, inaddition, the compartments of the frame may or may not be square areacompartments and may include rectangular compartments as shown. In someexamples, the area compartments of frame 68 may be evenly spaced havingsquare dimensions (e.g., prior to warping).

FIG. 8 is an example conceptual illustration 704 of a warped frame 78 ofimage data created using a warping grid 72. Camera processor(s) 14 orencoder/decoder 17 may apply warping grid 72 to frame 68 to createwarped frame 78. The indicated area 76 is also warped so as to be largerthan pre-warped indicated area 70 in the original frame 68. This isbecause camera processor(s) 14 or encoder/decoder 17 apply warping grid72 around indicated area 70 in order to allocate more area of the frameto the one or more indicated areas 76 while maintaining the same numberof pixels from frame 68 to warped frame 78. That is, frames 68 and 78are of the same size because both frames comprise a same number ofpixels. In addition, the areas adjacent the area of importance, such asarea compartment 74 is allocated less area (e.g., fewer pixels). Thatis, the warping of grid lines 72 in warped frame 78 illustrate thegeometric displacement of pixels of frame 78, such that area 76 isallocated more pixels compared to area 74.

In some examples, camera processor(s) 14 may perform offline processingof the warped frame. For example, camera processor(s) 14 may performspatial processing, temporal processing, sharpening, or denoising usingthe warped frame. In some examples, IPE 42B may perform offlineprocessing using warped frame 78. In any event, warped frame 78 may beoutput to an encoder/decoder 17 or system memory 30 for furtherprocessing. For example, encoder/decoder 17 may be configured to performscaling of warped frame , as described with reference to FIGS. 9 and 10,to achieve a desired output resolution. As such, camera processor(s) 14may provide better processing, quality, resolution, etc. of a frame ofimage data. For example, IPE 42B may process indicated areas (e.g.,areas of importance) with more pixels relative to areas adjacent theindicated areas as a result of the warped frame. In one example, IPE 42Bmay process the indicated area of importance with more processingresources because the frame warped around the indicated area causes theindicated area to be enlarged and thus, use more area of the framerelative to areas adjacent the indicated area. For example, theindicated area may appear, if depicted visually, as distorted andenlarged to proportional to the increase in pixels allocated theindicated area. That is, if twice the number of pixels were allocatedthe indicated area, the indicated area may be enlarged to twice the sizeif the warped frame were to be depicted visually. In such anillustrative example, the areas adjacent the indicated area may shrinkto half the original size, such that the frame does not increase in sizeprior to the inverse warping and scaling operation.

IPE 42B may process the one or more variously indicated areas withhigher resolution because IPE 42B may process more pixels for theindicated area(s) relative to shrunken areas of the frame adjacent theindicated area(s), the adjacent areas having fewer pixels allocatedrelative to the unwarped frame received from one of image sensor(s) 12.In any event, the warped frame may have a same number of total pixels asthe unwarped frame configuration.

A person of skill in the art would understand that although only oneobject of importance is shown to represent the area of importance, thearea of importance may comprise multiple connected or disconnected areasof the frame and may both be of amorphous shapes, hollow shapes, etc. Inaddition, while the conceptual warping grid 72 is shown as havingdisparate lines, the grid and warping grid may actually have sparse datapoints that may be as numerous as the number of pixels of imagesensor(s) 12. In some examples, the grid 66 may only be applied toportions of the frame, such as only to portions of the frame that wouldbe relevant after a digital cropping operation to achieve a desired zoomlevel. In some examples, camera processor(s) 14 may apply warping and/orinverse warping operations coupled with identity mapping operations. Forexample, camera processor(s) 14 may use identity mapping to maintainpixels at particular locations relative to a frame of image data. Insome examples, the warping operation may cause a displacement of pixelsaccording to input and output mapping coordinates of a warping frame. Insuch examples, camera processor(s) 14 may use identity mapping tocontrol the warping of an indicated area of a frame, such that no pixeldisplacement occurs for the indicated area of the frame during theunwarping process.

FIG. 9 is an example conceptual illustration of a warped frame 804undergoing a scaling and unwarping process to achieve a desired outputresolution. In some instances, the desired output resolution correspondsto an output resolution that utilizes more pixels relative to a numberof pixels output from image sensor(s) 12. For example, a 4K sensor maybe used to produce 8K video. In such instances, camera processor(s) 14may utilize upscaling to achieve the desired output resolution. Inaccordance with techniques of this disclosure, however, the upscalinglevels used to upscale various areas of warped frame 78 may be differentaccording to the area of each compartment formed by warped frame 78.

In the example of FIG. 9, the unwarped frame 806 is larger in size toillustrate the difference in output resolution from warped frame 804 tounwarped frame 806. In the illustrative example of FIG. 9, the unwarpedframe 806 is twice as big as warped frame 804 to illustrate thedifference in output resolution in one particular example, such as a 4Ksensor producing 8K video (˜2× difference in terms of pixel count). Assuch, the scaling process performed by camera processor(s) 14 orencoder/decoder 17 includes unwarping warped frame 804, such thatwarping is removed from the scaled output frame, such as is shown withunwarped frame 806.

In such examples, camera processor(s) 14 or encoder/decoder 17 mayperform an inverse distortion operation by applying an inverse warpinggrid 82 to warped frame 804 to achieve an unwarped frame 88 at thedesired output resolution. In performing the inverse distortionoperation, camera processor(s) 14 or encoder/decoder 17 may performscaling to frame 804 to upscale frame 804 to achieve frame 88 of agreater size. In a non-limiting and illustrative example, cameraprocessor(s) 14 or encoder/decoder 17 may perform scaling to frame 804to upscale frame 804 to achieve frame 88 of twice the size (e.g., twicethe display pixel resolution). In doing so, camera processor(s) 14 orencoder/decoder 17 may perform more upscaling using more pixels to theareas of the warped frame 804 comprising indicated area 86 and moredownscaling to areas adjacent the indicated area 84. As such, adjacentarea 84 may have more blur and/or distortion as a result of the higherscaling level (e.g., more downscaling) used for the adjacent areas, andother surrounding areas, compared to indicated area of importance 86.The indicated area may increase in size due to the upscaling but at alesser rate because the warped indicated area is enlarged as a result ofthe warping grid, as described with reference to FIG. 8.

In another example, scaling warped frame 804 to undistorted frame 88 mayinclude camera processor(s) 14 upscaling at a relatively higherupscaling level that may in some instances, include no scaling, inportions of warped frame 804 that have been warped around the previouslyindicated area of warped frame 804. In addition, camera processor(s) 14may downscale at a relatively higher downscaling level in portions ofwarped frame 804 corresponding to areas adjacent the indicated area ofwarped frame 804. As such, the indicated areas may be scaled at a firstscaling level and the areas adjacent may be scaled at a second scalinglevel, where the first scaling level (e.g., an upscaling ratio ornon-scaling ratio) may include a scaling level that is higher than thesecond scaling level.

In some examples, camera processor(s) 14 or encoder/decoder 17 mayperform scaling using a first scaling level with respect to pixelscorresponding to the indicated area. In some examples, the first scalinglevel may correspond to zero scaling or no scaling, such as a 1-to-1scaling ratio. In addition, camera processor(s) 14 or encoder/decoder 17may perform scaling using a second scaling level with respect to pixelscorresponding to at least one area adjacent the indicated area. In suchexamples, the second scaling level may be different from the firstscaling level. For example, the second scaling level may be greater thanthe first scaling level, such that more downscaling occurs with areasadjacent the indicated area and less downscaling occurs with theindicated area (e.g., prior to higher level processing). In someinstances, the levels may be proportional to the size of respectivecompartments after applying a warping grid.

FIG. 10 is an example conceptual illustration 908 of a warped frame 904undergoing a scaling process to achieve a desired output resolution. Forexample, camera processor(s) 14 may comply with limited processingresources of a camera system by downscaling an image capture. In oneexample, camera processor(s) 14 may have limited processing capabilitiesfor outputting high frame rates at high resolution and thus, maydownscale the image to achieve a desired frame rate. In the example ofFIG. 10, unwarped frame 98 may be smaller than warped frame 904 due to ascaling process. Camera processor(s) 14 perform scaling using a firstscaling level with respect to pixels corresponding to the one or moreindicated areas 96. Camera processor(s) 14 may perform downscaling usinga second scaling level with respect to pixels corresponding to at leastone area adjacent the indicated areas 94, where the second scaling levelis greater than the first scaling level. For example, the scaling levelsmay include more downscaling in areas adjacent the indicated areas 94,including other surrounding areas, and less upscaling, or no scaling insome instances, with respect to pixels corresponding to the one or moreindicated areas 96.

In some examples, encoder/decoder 17 may perform the scaling asencoder/decoder 17 reads input from camera processor(s) 14, such aswhile reading bits of data corresponding to warped frame 904. In anyevent, camera processor(s) 14 or encoder/decoder 17 may apply an inversewarping grid 92 to achieve unwarped frame 98.

A person of skill in the art would understand that image captures mayinclude a snapshot capture or a video stream capture. For example, oneor more cameras 15 of computing device 10 may implement the operationsshown in FIGS. 7-10 while recording video, providing an image preview(e.g., before a user takes a photo, still photos, pictures-in-video,motion photos, etc.).

In one or more examples, the functions described may be implemented inhardware, software, firmware, or any combination thereof. If implementedin software, the functions may be stored on or transmitted over, as oneor more instructions or code, a computer-readable medium and executed bya hardware-based processing unit. Computer-readable media may includecomputer-readable storage media, which corresponds to a tangible mediumsuch as data storage media. In this manner, computer-readable mediagenerally may correspond to tangible computer-readable storage mediawhich is non-transitory. Data storage media may be any available mediathat can be accessed by one or more computers or one or more processorsto retrieve instructions, code and/or data structures for implementationof the techniques described in this disclosure. A computer programproduct may include a computer-readable medium.

By way of example, and not limitation, such computer-readable storagemedia can comprise RAM, ROM, EEPROM, CD-ROM or other optical diskstorage, magnetic disk storage, or other magnetic storage devices, flashmemory, cache memory, or any other medium that can be used to storedesired program code in the form of instructions or data structures andthat can be accessed by a computer. It should be understood thatcomputer-readable storage media and data storage media do not includecarrier waves, signals, or other transient media, but are insteaddirected to non-transient, tangible storage media. Disk and disc, asused herein, includes compact disc (CD), laser disc, optical disc,digital versatile disc (DVD), floppy disk and Blu-ray disc, where discsusually reproduce data magnetically, while discs reproduce dataoptically with lasers. Combinations of the above should also be includedwithin the scope of computer-readable media.

Instructions may be executed by one or more processors, such as one ormore digital signal processors (DSPs), general purpose microprocessors,application specific integrated circuits (ASICs), field programmablelogic arrays (FPGAs), or other equivalent integrated or discrete logiccircuitry. Accordingly, the term “processor,” as used herein may referto any of the foregoing structure or any other structure suitable forimplementation of the techniques described herein. In addition, in someaspects, the functionality described herein may be provided withindedicated hardware and/or software modules configured for encoding anddecoding, or incorporated in a combined codec. Also, the techniquescould be fully implemented in one or more circuits or logic elements.

The techniques of this disclosure may be implemented in a wide varietyof devices or apparatuses, including a wireless handset, an integratedcircuit (IC) or a set of ICs (e.g., a chip set). Various components,modules, or units are described in this disclosure to emphasizefunctional aspects of devices configured to perform the disclosedtechniques, but do not necessarily require realization by differenthardware units. Rather, as described above, various units may becombined in a codec hardware unit or provided by a collection ofinteroperative hardware units, including one or more processors asdescribed above, in conjunction with suitable software and/or firmware.

Various examples have been described. These and other examples arewithin the scope of the following claims.

1. An apparatus configured for camera processing, the apparatuscomprising: a memory; and one or more processors in communication withthe memory, the one or more processors configured to: cause an imagesensor to receive a frame of image data in response to a user input;receive, from the image sensor, the frame, wherein the received frameincludes a first number of pixels based on an output resolution of theimage sensor; warp the received frame around a first area to create awarped frame; scale the warped frame to create a scaled frame, whereinthe scaled frame has more pixels than the first number of pixels; andoutput the scaled frame for processing.
 2. (canceled)
 3. The apparatusof claim 1, wherein the one or more processors are configured to: scaleusing a first scaling level with respect to pixels corresponding to thefirst area; and scale using a second scaling level with respect topixels corresponding to at least one area adjacent the first area,wherein the second scaling level is different from the first scalinglevel.
 4. The apparatus of claim 1, wherein the one or more processorsare further configured to: perform an inverse distortion operation thatapplies inverse warping to remove warping of the scaled frame to createan unwarped frame.
 5. (canceled)
 6. The apparatus of claim 4, whereinthe one or more processors are configured to output the unwarped frameto one or more of a video encoder, a display device, or a memory device.7. (canceled)
 8. The apparatus of claim 1, wherein the one or moreprocessors are further configured to: perform one or more of spatialprocessing, temporal processing, sharpening, or denoising on the scaledframe.
 9. The apparatus of claim 1, wherein the one or more processorsare further configured to: receive a subsequent frame of the image datacomprising at least a portion of the first area; track the first area tothe subsequent frame; warp the subsequent frame around the first area astracked to create a second warped frame; and output the second warpedframe.
 10. The apparatus of claim 9, wherein the one or more processorsare further configured to: perform no scaling with respect to pixelscorresponding to the first area in the second warped frame; and performdownscaling with respect to pixels corresponding to an area adjacent thefirst area in the second warped frame.
 11. The apparatus of claim 9,wherein the first area tracked to the subsequent frame includes at leasta portion of a face.
 12. (canceled)
 13. (canceled)
 14. The apparatus ofclaim 1, wherein the first area is identified based on a user selection.15. The apparatus of claim 1, wherein to receive indication of the firstarea is based on an automatic detection process.
 16. The apparatus ofclaim 1, wherein to warp the frame, the one or more processors areconfigured to: allocate more pixels to the first area and allocate fewerpixels to areas adjacent the first area.
 17. The apparatus of claim 16,wherein to warp the frame, the one or more processors are configured to:apply a warping grid to the frame, the frame including the first area,and the warping grid configured to cause a geometric displacement ofpixels relative to the received frame.
 18. The apparatus of claim 17,wherein to create the warped frame, the one or more processors areconfigured to: apply the warping grid to the frame including to thefirst area and to at least one area of the frame adjacent the firstarea, wherein application of the warping grid results in more area ofthe warped frame being allocated to the first area relative to an amountof area of the warped frame allocated to the at least one area of theframe adjacent the first area.
 19. A method for camera processing, themethod comprising: causing an image sensor to receive a frame of imagedata in response to a user input; receiving, from the image sensor, theframe, wherein the received frame includes a first number of pixelsbased on an output resolution of the image sensor; warping the receivedframe around a first area to create a warped frame; scaling the warpedframe to create a scaled frame, wherein the scaled frame has more pixelsthan the first number of pixels; and outputting the scaled frame forprocessing.
 20. The method of claim 19, the method further comprising:performing an inverse distortion operation that applies inverse warpingto remove warping of the scaled frame to create an unwarped frame. 21.(canceled)
 22. The method of claim 20, the method further comprising:outputting the unwarped frame to one or more of a video encoder, adisplay device, or a memory device.
 23. The method of claim 19, themethod further comprising: performing one or more of spatial processing,temporal processing, sharpening, or denoising on the scaled frame. 24.The method of claim 19, the method further comprising: performingscaling using a first scaling level with respect to pixels correspondingto the first area; and performing scaling using a second scaling levelwith respect to pixels corresponding to at least one area adjacent thefirst area, wherein the second scaling level is different from the firstscaling level.
 25. The method of claim 19, the method furthercomprising: receiving a subsequent frame of the image data comprising atleast a portion of the first area; tracking the first area to thesubsequent frame; warping the subsequent frame around the first area astracked to create a second warped frame; and outputting the secondwarped frame.
 26. An apparatus configured for camera processing, theapparatus comprising: means for receiving a frame of image dataincluding a first number of pixels based on an output resolution of animage sensor; means for warping the frame around a first area to createa warped frame; means for scaling the warped frame to create a scaledframe, wherein the scaled frame has more pixels than the first number ofpixels; and means for outputting the scaled frame for processing. 27.(canceled)
 28. The apparatus of claim 27, the apparatus furthercomprising: means for outputting the scaled frame to one or more of avideo encoder, a display device, or a memory device.
 29. Anon-transitory computer-readable storage medium having stored thereoninstructions that, when executed, cause one or more processors to: causean image sensor to receive a frame of image data in response to a userinput; receive, from the image sensor, the frame, wherein the receivedframe includes a first number of pixels based on an output resolution ofthe image sensor; warp the received frame around a first area to createa warped frame; scale the warped frame to produce a scaled frame,wherein the scaled frame has more pixels than the first number ofpixels; and output the scaled frame for processing.
 30. Thenon-transitory computer-readable storage medium of claim 29, wherein theone or more processors are further caused to: perform an inversedistortion operation that applies inverse warping to remove warping ofthe scaled frame to create an unwarped frame.
 31. The apparatus of claim1, wherein the one or more processors are further configured to:determine one or more of a focus, exposure, or white balance settingbased on the scaled frame.
 32. The method of claim 19, furthercomprising determining one or more of a focus, exposure, or whitebalance setting based on the scaled frame.